license: “CC BY-NC”
Creative Commong: Attribution, Non-Commerical
https://creativecommons.org/licenses/by-nc/4.0/

Find this repository: https://github.com/libjohn/workshop_twitter_analysis

Much of this review comes from Introduction to gathering tweets with rwteet using the rtweet package. Conveniently, you no longer need a Twitter API developer key to use this package. You do need a Twitter account.

How to gather twitter data via the API

Load library packages

Use the tidyverse and rtweet

library(tidyverse)
library(rtweet)

Search tweets

search_tweets() - search a keyword(s) or hashtag

I recommend limiting the number of tweets returned (n = 1000) for this training. Otherwise you may hit a rate limit.

#bts <- search_tweets("#BTS", n = 5000, include_rts = FALSE)
bts_dynamite <- search_tweets("#BTS dynamite", n = 1000, include_rts = FALSE)
Requesting token on behalf of user...
Waiting for authentication in browser...
Press Esc/Ctrl + C to abort
Authentication complete.

Downloading [=======>---------------------------------]  20%
Downloading [===========>-----------------------------]  30%
Downloading [===============>-------------------------]  40%
Downloading [===================>---------------------]  50%
Downloading [========================>----------------]  60%
Downloading [============================>------------]  70%
Downloading [================================>--------]  80%
Downloading [====================================>----]  90%
Downloading [=========================================] 100%

Gathering tweets will return a 90 variable tibble of whatever row-size you were able to collect. You’ll want to spend some time familiarizing yourself with these variables and the range of data that can be gathered.

# bts
bts_dynamite

Serialize the gathering

get friends

Find all the accounts a user follows

john_little <-  get_friends("john_little")

This returns the twitter name, i.e. user, and the user_id for each person following that user.

john_little

Next, use lookup_users with the user_id to get more information about those accounts.

john_little_data <- lookup_users(john_little$user_id)
john_little_data

get followers

Who is following me? get_followers()

jrl_flw <- get_followers("john_little")
jrl_flw_data <- lookup_users(jrl_flw$user_id)
jrl_flw_data 

timelines

Get the most recent tweets from an account

rg_tmls <- get_timelines("RhiannonGiddens", n = 3200)

Visuzlize

rg_tmls %>% 
  dplyr::filter(created_at >= "2016-01-01") %>%  
  dplyr::group_by(screen_name) %>%
  ts_plot("weeks", trim = 1L) +
  ggplot2::geom_point() +
  geom_smooth(se = FALSE, color = "cadetblue") +
  colorblindr::scale_color_OkabeIto() +
  hrbrthemes::theme_ipsum(grid = "Y") +
  ggplot2::theme(
    legend.title = ggplot2::element_blank(),
    legend.position = "bottom", 
    plot.title = ggplot2::element_text(face = "bold")
    ) +
    ggplot2::labs(
    x = NULL, y = NULL,
    title = "Frequency of Twitter statuses",
    subtitle = "Twitter status (tweet) counts aggregated by week from Jan. 2016",
    caption = "Source: Data collected from Twitter's REST API via rtweet"
  )

NA
NA

get_favorites

Get the most recent favorites from a user

rg_faves <- get_favorites("RhiannonGiddens", n = 3000)
rg_faves

Profiles

Search a users’ profiles

gullah <- search_users("#gullah", n = 1000)
Searching for users...
Finished collecting users!
gullah

Location information

Using the tidygeocoder R library, we can find location information when place_names are available.

First, geocode

Use the tidygeocoder package.

# glimpse(rg_tmls)
rg_places <- rg_tmls %>% 
  drop_na(place_name) %>% 
  select(place_name:bbox_coords) %>% 
  distinct() %>% 
  mutate(addr = glue::glue("{place_full_name}, {country}")) %>% 
  tidygeocoder::geocode(addr, method = "osm")

rg_places

Visuzlize

You can create maps in R. Below is one of the easiest methods, especially if you know ggplot2

rg_places %>% 
  distinct() %>% 
  drop_na(lat) %>% 
  ggplot(aes(long, lat), color="grey99") +
  borders("world") + 
  geom_point(color = "goldenrod") + 
  ggrepel::geom_label_repel(aes(label = place_full_name), 
                            segment.color = "goldenrod", segment.size = 1,
                            color = "navy") + 
  theme_void()

Second location example

Very similar to above. For accounts with “#gullah” in their profile, and that have location information listed, geocode the locations …..

gullah_places <- gullah %>% 
  drop_na(place_name) %>% 
  select(place_name:bbox_coords) %>% 
  filter(country_code == "US")  %>%
  distinct() %>% 
  mutate(addr = glue::glue("{place_full_name}, {country}")) %>% 
  tidygeocoder::geocode(addr, method = "osm")

gullah_places

And now visualize on a US map of the lower 48 states.

You can learn more about basic R mapping from our workshop on mapping with R

gullah_places %>% 
  distinct() %>% 
  drop_na(lat) %>% 
  ggplot(aes(long, lat), color="grey99") +
  borders("state") + 
  geom_point(color = "goldenrod") + 
  ggrepel::geom_label_repel(aes(label = place_full_name), 
                            segment.color = "goldenrod", segment.size = 1,
                            color = "navy") + 
  theme_void()

LS0tDQp0aXRsZTogImdhdGhlciB3aXRoIHJ0d2VldCINCmF1dGhvcjogIkpvaG4gTGl0dGxlIg0KZGF0ZTogImByIFN5cy5EYXRlKClgIg0Kb3V0cHV0OiBodG1sX25vdGVib29rDQotLS0NCg0KbGljZW5zZTogIkNDIEJZLU5DIiAgDQpDcmVhdGl2ZSBDb21tb25nOiAgQXR0cmlidXRpb24sIE5vbi1Db21tZXJpY2FsICANCmh0dHBzOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS1uYy80LjAvICANCg0KDQpGaW5kIHRoaXMgcmVwb3NpdG9yeTogIGh0dHBzOi8vZ2l0aHViLmNvbS9saWJqb2huL3dvcmtzaG9wX3R3aXR0ZXJfYW5hbHlzaXMNCg0KTXVjaCBvZiB0aGlzIHJldmlldyBjb21lcyBmcm9tIFtJbnRyb2R1Y3Rpb24gdG8gZ2F0aGVyaW5nIHR3ZWV0cyB3aXRoIHJ3dGVldF0oaHR0cHM6Ly9kb2NzLnJvcGVuc2NpLm9yZy9ydHdlZXQvYXJ0aWNsZXMvaW50cm8uaHRtbCkgdXNpbmcgdGhlIFtgcnR3ZWV0YCBwYWNrYWdlXShodHRwczovL2RvY3Mucm9wZW5zY2kub3JnL3J0d2VldC8pLiAgQ29udmVuaWVudGx5LCB5b3UgKipubyBsb25nZXIgbmVlZCoqIGEgW1R3aXR0ZXIgQVBJIGRldmVsb3BlciBrZXldKGh0dHBzOi8vZG9jcy5yb3BlbnNjaS5vcmcvcnR3ZWV0L2FydGljbGVzL2F1dGguaHRtbCkgdG8gdXNlIHRoaXMgcGFja2FnZS4gIFlvdSAqKmRvIG5lZWQgYSBUd2l0dGVyIGFjY291bnQuKiogIA0KDQo8Y2VudGVyPioqSG93IHRvIGdhdGhlciB0d2l0dGVyIGRhdGEgdmlhIHRoZSBBUEkqKjwvY2VudGVyPg0KDQojIyBMb2FkIGxpYnJhcnkgcGFja2FnZXMNCg0KVXNlIHRoZSBbdGlkeXZlcnNlXShodHRwczovL3RpZHl2ZXJzZS5vcmcpIGFuZCBydHdlZXQNCg0KYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCmxpYnJhcnkodGlkeXZlcnNlKQ0KbGlicmFyeShydHdlZXQpDQpgYGANCg0KDQojIyBTZWFyY2ggdHdlZXRzDQoNCmBzZWFyY2hfdHdlZXRzKClgIC0gc2VhcmNoIGEga2V5d29yZChzKSBvciBoYXNodGFnDQoNCkkgcmVjb21tZW5kIGxpbWl0aW5nIHRoZSBudW1iZXIgb2YgdHdlZXRzIHJldHVybmVkIChgbiA9IDEwMDBgKSBmb3IgdGhpcyB0cmFpbmluZy4gIE90aGVyd2lzZSB5b3UgbWF5IGhpdCBhIHJhdGUgbGltaXQuDQoNCmBgYHtyfQ0KI2J0cyA8LSBzZWFyY2hfdHdlZXRzKCIjQlRTIiwgbiA9IDUwMDAsIGluY2x1ZGVfcnRzID0gRkFMU0UpDQpidHNfZHluYW1pdGUgPC0gc2VhcmNoX3R3ZWV0cygiI0JUUyBkeW5hbWl0ZSIsIG4gPSAxMDAwLCBpbmNsdWRlX3J0cyA9IEZBTFNFKQ0KYGBgDQoNCkdhdGhlcmluZyB0d2VldHMgd2lsbCByZXR1cm4gYSA5MCB2YXJpYWJsZSB0aWJibGUgb2Ygd2hhdGV2ZXIgcm93LXNpemUgeW91IHdlcmUgYWJsZSB0byBjb2xsZWN0LiAgWW91J2xsIHdhbnQgdG8gc3BlbmQgc29tZSB0aW1lIGZhbWlsaWFyaXppbmcgeW91cnNlbGYgd2l0aCB0aGVzZSB2YXJpYWJsZXMgYW5kIHRoZSByYW5nZSBvZiBkYXRhIHRoYXQgY2FuIGJlIGdhdGhlcmVkLg0KDQpgYGB7cn0NCiMgYnRzDQpidHNfZHluYW1pdGUNCmBgYA0KDQojIyBTZXJpYWxpemUgdGhlIGdhdGhlcmluZw0KDQotIFdpbmRvd3M6ICBodHRwczovL2dpdGh1Yi5jb20vYm5vc2FjL3Rhc2tzY2hlZHVsZVINCi0gTGludXg6IGh0dHBzOi8vZ2l0aHViLmNvbS9ibm9zYWMvY3JvblIjY3JvbnINCg0KIyMgZ2V0IGZyaWVuZHMNCg0KRmluZCBhbGwgdGhlIGFjY291bnRzIGEgdXNlciBmb2xsb3dzDQoNCmBgYHtyfQ0Kam9obl9saXR0bGUgPC0gIGdldF9mcmllbmRzKCJqb2huX2xpdHRsZSIpDQpgYGANCg0KVGhpcyByZXR1cm5zIHRoZSB0d2l0dGVyIG5hbWUsIGkuZS4gYHVzZXJgLCBhbmQgdGhlIGB1c2VyX2lkYCBmb3IgZWFjaCBwZXJzb24gZm9sbG93aW5nIHRoYXQgdXNlci4NCg0KYGBge3J9DQpqb2huX2xpdHRsZQ0KYGBgDQoNCk5leHQsIHVzZSBgbG9va3VwX3VzZXJzYCB3aXRoIHRoZSBgdXNlcl9pZGAgdG8gZ2V0IG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgdGhvc2UgYWNjb3VudHMuDQoNCmBgYHtyfQ0Kam9obl9saXR0bGVfZGF0YSA8LSBsb29rdXBfdXNlcnMoam9obl9saXR0bGUkdXNlcl9pZCkNCmBgYA0KDQpgYGB7cn0NCmpvaG5fbGl0dGxlX2RhdGENCmBgYA0KDQojIyBnZXQgZm9sbG93ZXJzDQoNCldobyBpcyBmb2xsb3dpbmcgbWU/ICBgZ2V0X2ZvbGxvd2VycygpYA0KDQpgYGB7cn0NCmpybF9mbHcgPC0gZ2V0X2ZvbGxvd2Vycygiam9obl9saXR0bGUiKQ0KYGBgDQoNCg0KYGBge3J9DQpqcmxfZmx3X2RhdGEgPC0gbG9va3VwX3VzZXJzKGpybF9mbHckdXNlcl9pZCkNCmBgYA0KDQoNCmBgYHtyfQ0KanJsX2Zsd19kYXRhIA0KYGBgDQoNCiMjIHRpbWVsaW5lcw0KDQpHZXQgdGhlIG1vc3QgcmVjZW50IHR3ZWV0cyBmcm9tIGFuIGFjY291bnQNCg0KYGBge3J9DQpyZ190bWxzIDwtIGdldF90aW1lbGluZXMoIlJoaWFubm9uR2lkZGVucyIsIG4gPSAzMjAwKQ0KYGBgDQoNCg0KYGBge3J9DQpyZ190bWxzICU+JSANCiAgc3VtbWFyaXNlKG1pbihjcmVhdGVkX2F0KSwgbWF4KGNyZWF0ZWRfYXQpKQ0KYGBgDQoNCiMjIyBWaXN1emxpemUNCg0KYGBge3Igd2FybmluZz1GQUxTRX0NCnJnX3RtbHMgJT4lIA0KICBkcGx5cjo6ZmlsdGVyKGNyZWF0ZWRfYXQgPj0gIjIwMTYtMDEtMDEiKSAlPiUgIA0KICBkcGx5cjo6Z3JvdXBfYnkoc2NyZWVuX25hbWUpICU+JQ0KICB0c19wbG90KCJ3ZWVrcyIsIHRyaW0gPSAxTCkgKw0KICBnZ3Bsb3QyOjpnZW9tX3BvaW50KCkgKw0KICBnZW9tX3Ntb290aChzZSA9IEZBTFNFLCBjb2xvciA9ICJjYWRldGJsdWUiKSArDQogIGNvbG9yYmxpbmRyOjpzY2FsZV9jb2xvcl9Pa2FiZUl0bygpICsNCiAgaHJicnRoZW1lczo6dGhlbWVfaXBzdW0oZ3JpZCA9ICJZIikgKw0KICBnZ3Bsb3QyOjp0aGVtZSgNCiAgICBsZWdlbmQudGl0bGUgPSBnZ3Bsb3QyOjplbGVtZW50X2JsYW5rKCksDQogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsIA0KICAgIHBsb3QudGl0bGUgPSBnZ3Bsb3QyOjplbGVtZW50X3RleHQoZmFjZSA9ICJib2xkIikNCiAgICApICsNCiAgICBnZ3Bsb3QyOjpsYWJzKA0KICAgIHggPSBOVUxMLCB5ID0gTlVMTCwNCiAgICB0aXRsZSA9ICJGcmVxdWVuY3kgb2YgVHdpdHRlciBzdGF0dXNlcyIsDQogICAgc3VidGl0bGUgPSAiVHdpdHRlciBzdGF0dXMgKHR3ZWV0KSBjb3VudHMgYWdncmVnYXRlZCBieSB3ZWVrIGZyb20gSmFuLiAyMDE2IiwNCiAgICBjYXB0aW9uID0gIlNvdXJjZTogRGF0YSBjb2xsZWN0ZWQgZnJvbSBUd2l0dGVyJ3MgUkVTVCBBUEkgdmlhIHJ0d2VldCINCiAgKQ0KDQoNCmBgYA0KDQojIyBnZXRfZmF2b3JpdGVzDQoNCkdldCB0aGUgbW9zdCByZWNlbnQgZmF2b3JpdGVzIGZyb20gYSB1c2VyDQoNCmBgYHtyfQ0KcmdfZmF2ZXMgPC0gZ2V0X2Zhdm9yaXRlcygiUmhpYW5ub25HaWRkZW5zIiwgbiA9IDMwMDApDQpgYGANCg0KYGBge3J9DQpyZ19mYXZlcw0KYGBgDQoNCiMjIFByb2ZpbGVzDQoNClNlYXJjaCBhIHVzZXJzJyBwcm9maWxlcw0KDQpgYGB7cn0NCmd1bGxhaCA8LSBzZWFyY2hfdXNlcnMoIiNndWxsYWgiLCBuID0gMTAwMCkNCmBgYA0KDQoNCmBgYHtyfQ0KZ3VsbGFoDQpgYGANCg0KIyMgZ2V0IHRyZW5kcw0KDQpXaGF0IGlzIHRyZW5kaW5naW4gYSBzcGVjaWZpYyBsb2NhdGlvbj8NCg0KYGBge3J9DQojIHNmIDwtIGdldF90cmVuZHMoInNhbiBmcmFuY2lzbyIpDQojIGR1cmhhbSA8LSBnZXRfdHJlbmRzKGxhdCA9IDM2LjAsIGxuZyA9IC03OC45KQ0KZ3JlZW5zYm9ybyA8LSBnZXRfdHJlbmRzKCJncmVlbnNib3JvIikNCmBgYA0KDQpgYGB7cn0NCmdyZWVuc2Jvcm8NCmBgYA0KDQojIyBMb2NhdGlvbiBpbmZvcm1hdGlvbg0KDQpVc2luZyB0aGUgdGlkeWdlb2NvZGVyIFIgbGlicmFyeSwgd2UgY2FuIGZpbmQgbG9jYXRpb24gaW5mb3JtYXRpb24gd2hlbiBwbGFjZV9uYW1lcyBhcmUgYXZhaWxhYmxlLg0KDQojIyMgRmlyc3QsIGdlb2NvZGUgDQoNClVzZSB0aGUgW3RpZHlnZW9jb2Rlcl0oaHR0cHM6Ly9qZXNzZWNhbWJvbi5naXRodWIuaW8vdGlkeWdlb2NvZGVyKSBwYWNrYWdlLiAgDQoNCmBgYHtyfQ0KIyBnbGltcHNlKHJnX3RtbHMpDQpyZ19wbGFjZXMgPC0gcmdfdG1scyAlPiUgDQogIGRyb3BfbmEocGxhY2VfbmFtZSkgJT4lIA0KICBzZWxlY3QocGxhY2VfbmFtZTpiYm94X2Nvb3JkcykgJT4lIA0KICBkaXN0aW5jdCgpICU+JSANCiAgbXV0YXRlKGFkZHIgPSBnbHVlOjpnbHVlKCJ7cGxhY2VfZnVsbF9uYW1lfSwge2NvdW50cnl9IikpICU+JSANCiAgdGlkeWdlb2NvZGVyOjpnZW9jb2RlKGFkZHIsIG1ldGhvZCA9ICJvc20iKQ0KDQpyZ19wbGFjZXMNCmBgYA0KDQojIyMgVmlzdXpsaXplIA0KDQpZb3UgY2FuIGNyZWF0ZSBtYXBzIGluIFIuICBCZWxvdyBpcyBvbmUgb2YgdGhlIGVhc2llc3QgbWV0aG9kcywgZXNwZWNpYWxseSBpZiB5b3Uga25vdyBbZ2dwbG90Ml0oaHR0cHM6Ly9nZ3Bsb3QyLnRpZHl2ZXJzZS5vcmcpDQoNCmBgYHtyfQ0KcmdfcGxhY2VzICU+JSANCiAgZGlzdGluY3QoKSAlPiUgDQogIGRyb3BfbmEobGF0KSAlPiUgDQogIGdncGxvdChhZXMobG9uZywgbGF0KSwgY29sb3I9ImdyZXk5OSIpICsNCiAgYm9yZGVycygid29ybGQiKSArIA0KICBnZW9tX3BvaW50KGNvbG9yID0gImdvbGRlbnJvZCIpICsgDQogIGdncmVwZWw6Omdlb21fbGFiZWxfcmVwZWwoYWVzKGxhYmVsID0gcGxhY2VfZnVsbF9uYW1lKSwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgc2VnbWVudC5jb2xvciA9ICJnb2xkZW5yb2QiLCBzZWdtZW50LnNpemUgPSAxLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9yID0gIm5hdnkiKSArIA0KICB0aGVtZV92b2lkKCkNCmBgYA0KDQojIyMgU2Vjb25kIGxvY2F0aW9uIGV4YW1wbGUNCg0KVmVyeSBzaW1pbGFyIHRvIGFib3ZlLiAgRm9yIGFjY291bnRzIHdpdGggIiNndWxsYWgiIGluIHRoZWlyIHByb2ZpbGUsIGFuZCB0aGF0IGhhdmUgbG9jYXRpb24gaW5mb3JtYXRpb24gbGlzdGVkLCBnZW9jb2RlIHRoZSBsb2NhdGlvbnMgLi4uLi4NCg0KYGBge3J9DQpndWxsYWhfcGxhY2VzIDwtIGd1bGxhaCAlPiUgDQogIGRyb3BfbmEocGxhY2VfbmFtZSkgJT4lIA0KICBzZWxlY3QocGxhY2VfbmFtZTpiYm94X2Nvb3JkcykgJT4lIA0KICBmaWx0ZXIoY291bnRyeV9jb2RlID09ICJVUyIpICAlPiUNCiAgZGlzdGluY3QoKSAlPiUgDQogIG11dGF0ZShhZGRyID0gZ2x1ZTo6Z2x1ZSgie3BsYWNlX2Z1bGxfbmFtZX0sIHtjb3VudHJ5fSIpKSAlPiUgDQogIHRpZHlnZW9jb2Rlcjo6Z2VvY29kZShhZGRyLCBtZXRob2QgPSAib3NtIikNCg0KZ3VsbGFoX3BsYWNlcw0KYGBgDQoNCkFuZCBub3cgdmlzdWFsaXplIG9uIGEgVVMgbWFwIG9mIHRoZSBfbG93ZXIgNDhfIHN0YXRlcy4NCg0KWW91IGNhbiBsZWFybiBtb3JlIGFib3V0IGJhc2ljIFIgbWFwcGluZyBmcm9tIG91ciB3b3Jrc2hvcCBvbiBbbWFwcGluZyB3aXRoIFJdKGh0dHBzOi8vcmZ1bi5saWJyYXJ5LmR1a2UuZWR1L3BvcnRmb2xpby9tYXBwaW5nX3dvcmtzaG9wLykNCg0KDQpgYGB7cn0NCmd1bGxhaF9wbGFjZXMgJT4lIA0KICBkaXN0aW5jdCgpICU+JSANCiAgZHJvcF9uYShsYXQpICU+JSANCiAgZ2dwbG90KGFlcyhsb25nLCBsYXQpLCBjb2xvcj0iZ3JleTk5IikgKw0KICBib3JkZXJzKCJzdGF0ZSIpICsgDQogIGdlb21fcG9pbnQoY29sb3IgPSAiZ29sZGVucm9kIikgKyANCiAgZ2dyZXBlbDo6Z2VvbV9sYWJlbF9yZXBlbChhZXMobGFiZWwgPSBwbGFjZV9mdWxsX25hbWUpLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZWdtZW50LmNvbG9yID0gImdvbGRlbnJvZCIsIHNlZ21lbnQuc2l6ZSA9IDEsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3IgPSAibmF2eSIpICsgDQogIHRoZW1lX3ZvaWQoKQ0KYGBgDQoNCg0K